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(54) Colour image pickup device 

(57) Apparatus (10.20.30) for recording an image, 
includes a two-dimensional array of image sensors. 
Each image sensor provides a measurement of the light 
intensity in a selected spectral region. The two-dimen- 
sional array is generated from a plurality of identical 
blocks (1 2,22.32) of sensors, the blocks (1 2.22,32) be- 
ing juxtaposed to form the array. Each of the blocks 
(12,22,32) has equal numbers of sensors for each of the 
spectral regions, the number of different spectral re- 
gions being at least three. The sensors in the blocks 
(1 2,22,32) are arranged in a two-dimensional array hav- 
ing a plurality of rows and columns. The sensors in the 
blocks (12.22.32) are arranged such that any straight 
line passing through a given sensor also passes through 
sensors of at least three different colors whose spectral 
responses are all linearly independent. All three such 
sensors may lie within a disk, centered at the first sensor 
and having a radius no larger than five times the center 
to center spacing of the blocks (1 2.22.32) in the sensor 
array Each of the sensors corresponding to one of the 
selected spectral regions in blocks (12=22,32) not adja- 
cent to an edge of the two-dimensional array may be 
adjacent a sensor corresponding to the same spectral 
region. 
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Description 

[CX)01] The present invention relates to apparatus for 
recording an image, for example for digital cameras and 
for an improved design for a color sensor array for use 5 
in digital cameras and the like. ' 
[0002] A digital color image usually consists of an ar- 
ray of pixel values representing the Intensity of the im- 
age at each point on a regular grid. Typically, three 
colors are used to generate the image. At each point on 
the grid the intensity of each pf' these colors is specified, 
thereby specifying both th^fhtensity and color of the im- 
age at that grid point. 

[0003] Conventional color photography records the 
relevant image data by utilizing three overlapping color 
sensing layers having sensitivities in different regions of 
the spectrum {usually red, green, and blue). Digital cam- 
eras, in contrast, typically utilize one array of sensors in 
a single "layer*. 

[0004] When only one sensor array is used to detect 
color images, only one color may be detected at any giv- 
en sensor location. As a result, these sensors do not 
produce a color image in the traditional sense, but rather 
a collection of individual color samples, which depend 
upon the assignment of color filters to individual sen- 
sors. This assignment is referred to as the color filter 
array (CFA) or the color mosaic pattern. To produce a 
true color image, with a full set of color samples (usually 
red, green and blue) at each sampling location, a sub- 
stantial amount of computation is required to estimate 
the missing information, since only a single color was 
originally sensed at each location in the array. 
[0005] There are a wide variety of approaches to the 
reconstruction problem, which is frequently referred to 
as the 'demosaicjng" task. The performance of all such 
algorithms depends upon the nature of the underlying 
CFA pattern. One such pattern is taught in U.S. Patent 
3,971,065. This pattern is generated by a repeating 2 x 
2 kernel, containing two green sensors, one red sensor 
and one blue sensor, as illustrated below: 



CFA 


GBGBGBGB 


Kernel 


G 


B 




RGRGRGRG 




B 


R 




G BGBG BGB 










RGRGRGRG 










GBGBGBGB 










RGRGRGRG 









[0006] It should be noted that this pattern has twice 
as many sensors in the green region of the spectrum as 
it does in the blue or red regions of the spectrum. The 
proponents of this pattern justify this choice on the 
grounds that the Human Visual System (HVS) is more 
sensitive to green. In addition, the proponents of this 
scheme point to the fact that it leads to relatively simple 
demosaicing algorithms. Specifically, if one color chan- 
nel is sampled more densely than the rest, it can be in- 



terpolated most easily and then used to guide the inter- 
polation of the less densely sampled color channels. 
[0007] The argument that the HVS is more sensitive 
to the green portion of the spectrum is fundamentally 
flawed when applied to digital camera systems; The rea- 
son the human eye can afford to have more green cones 
than red cones, and many fewer blue cones, is that the 
eye's optics effectively low-pass filter the red and blue 
parts of the spectrum more severely than the green. 
Hence, the eye cannot effectively utilize additional sam- 
ples in the red and blue regions of the spectrum. In ef- 
fect, the eye has poor spatial resolution for images in 
the red and blue regions of the spectrum. 
[0008] Digital cameras, in contrast, do not suffer from 
this limitation: hence, there is no logical reason for lim- 
iting the sampling density at any wavelength. If the sam- 
pling density in any of the color channels is reduced, 
there will be an increased ambiguity between spatial in- 
tensity variations and color composition of the original 
scene. Since such ambiguities are to be avoided, sam- 
pling densities in each of the colors should be the same. 
[0009] Another problem with the small 2x2 kernel de- 
scribed above lies in its inability to discern color and spa- 
tial variations on image components whose spatial di- 
mensions are small compared to the kernel's size. Con- 
sider a narrow red line (less than one sensor in width) 
running vertically through the image. If the line is posi- 
tioned such that it is over the column of sensors having 
only blue and green sensors, the system will have diffi- 
culty detecting the tine In fact, if neither the blue or 
green sensors have at least some response in the red 
region of the spectrum, the line cannot be detected at 
all. Needless to say. accurately determining the color of 
this line would be difficult. 

[0010] Small kernels also can present problems in 
demosaicing scenes having textures. Consider an im- 
age feature that is on the order of the size of the kernel. 
Demosaicing algorithms have difficulty determining 
whether the resultant sensor pattern corresponds to a 
smooth area in the scene with saturated color or a tex- 
tured region having rapidly varying intensity of a more 
neutral cotor. 

[0011] Hence, the size of the kernel should be rela- 
tively large, to assure that scene textures do not give 
the appearance of color variations. It should be noted, 
however, that there is a limit on the size of the kernel. 
As the kernel size grows, the implementation cost of the 
dennosaicing algorithm typically increases. Hence, a 
tradeoff between cost and image quality is generally im- 
plied. 

[0012] It should also be noted that images of interest 
to humans often include linear edges as well as lines. 
Hence, the sensor pattern should be designed so as to 
ensure that any linear feature in the image always pass- 
es through sensors of every color type. Furthermore, the 
distance along the linear feature between color sensors 
of the same type should be as small as possible. Failure 
to satisfy this requirement results in color ambiguity at 
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the edges of objects in the image. 
[001 3] The 2x2 pattern described above is clearly a 
poor choice in light of these requirements. First, it has 
a very small kernel. Second, it undersamples two of the 
color channels. Third, the red and blue sensors are nev- s 
er adjacent to sensors of the same color. Finally, hori- 
zontal and vertical linear features never pass through 
all three of the sensor types. 

[0014] One prior art solution to the above problems 
utilizes large kernels contairiirig pseudo-random distri- io 
butions of the different s^sor colors to minimize the 
likelihood that textures will induce colors in the demo- 
saiced image. However, such pseudo-random patterns 
introduce other problems. In addition to increasing the 
computational costs of demosaicing the image, there '5 
are regions in which there is too great a distance be- 
tween color sensors of the same color in a region of the 
image. This can make it difficult for the demosaicing al- 
gorithm to distinguish between spatial intensity varia- 
tions and color variations when it attempts to reconstruct 20 
a full cotor image. 

[0015] ■ The present invention seeks to provide im- 
proved image recording. 

[0016] According to an aspect of the present invention 
there is provided apparatus for recording an innage as 2S 
specified in claim 1. 

[0017] The preferred embodiment provides an im- 
proved color image sensor array. It can also provide an 
image sensor based on a kernel that is larger than the 
2x2 kernel described above. Preferably linear features 30 
pass through sensors for at least three different colors, 
whose spectral responses are linearly independent. Ad- 
vantageously, color sensors for the same color are lo- 
cated adjacent one another to minimize ambiguities in- 
troduced by rapidly changing intensity patterns in the im- 35 
age. 

[0018] The preferred embodiment provides appara- 
tus for recording an image which includes a two-dimen- 
sional array of image sensors. Each image sensor pro- 
vides a measurement of the light intensity in a selected ^0 
spectral region. The two-dimensional array is generated 
from a plurality of identical blocks of sensors, the blocks 
being juxtaposed to form the array Each of the blocks 
has equal numbers of sensors for each of the spectral 
regions, the number of different spectral regions being 
at least three, The sensors in the blocks are arranged 
in a two-dimensional array having a plurality of rows and 
columns. The sensors in the blocks are arranged such 
that any straight line passing through a given sensor al- 
so passes through sensors of at least three different so 
colors whose spectral responses are all linearly inde- 
pendent. 

[0019] In the preferred embodiment, all three such 
sensors lie within a disk, centered at the first sensor and 
having a radius no larger than five times the center to ss 
center spacing of the blocks in the sensor array In the 
preferred embodiment, each of the sensors correspond- 
ing to one of the selected spectral regions in blocks not 
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adjacent to an edge of the two-dimensional array is ad- 
jacent to a sensor corresponding to the same spectral 
region. 

[0020] An embodiment of the present invention is de- 
scribed below, by way of example only, with reference 
to the accompanying drawings, in which: 

Fig. 1 is a plan view of a preferred embodiment of 
image sensor; 

Fig. 2 is a plan view of^nother embodiment of image 
sensor; and 

Fig.3 is a plan view of a third embodiment of image 
sensor. 

[0021] Refer now to Figure 1 which is a plan view of 
an emtjodiment of color image sensor 1 0 which is based 
on a 4 X 3 replicating kernel, consisting of horizontally 
shifted rows of alternating red, green and blue sensors, 
as depicted at 12. 

[0022] The CFA pattern utilized in sensor 10 is a zig- 
zag pattern in which each of the red. green and blue 
channels may be traced in a zigzag fashion from the top 
of the array to the bottom. A portion of the zigzag pattern 
for one set of green sensors is shown at 1 3. The zigzag 
nature ensures that linear image features cannot pass 
through a subset of the sensors, which excludes one or 
more of the color channels. 

[0023] It should also be noted thai a sensor for one of 
the colors is adjacent to two other sensors of that color 
throughout the CFA for pixels away from the edges of 
the array This feature of the pattern makes it easier for 
the demosaicing algorithm to determine if a rapidly 
changing feature is the result of a change in color or a 
change in intensity (i.e.. a texture). 
[0024] Other members of the same class of zigzag 
patterns may be generated by adjusting the size and ar- 
rangement of the rows of the kernel to alter the nature 
of the zigzag. For example, the pattern might shift to the 
right for five consecutive rows, before it reverses and 
shifts to the left for the next five rows and so on. A sensor 
with this pattern is shown in Figure 2 at 20. Sensor 20 
is constructed from a 3x8 kernel shown at 22. The zig- 
zag pattern has linear segments that extend for 5 pixels 
before changing direction as shown at 23. 
[0025] In general, the pattern need not be a zigzag 
and the number of color sensors is not restricted to 
three. For example, a sensor based on a 
CMYG (cyan, magenta, yellow, and green) color 
scheme is shown in Figure 3 at 30. The kernel in this 
case is a 3x4-pixel array shown at 32. This pattern shifts 
to the right for 3 consecutive rows, and then jumps two 
rows to the left to repeat itself. The zigzag patterns and 
this latter pattern have in common the property that they 
are all generated by cyclic permutations of a row con- 
taining equal numbers of all color sensors. It will be clear 
to those skilled in the art from the above discussion that 
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he same strategy may be applied to a column;/ 
[0026] These patterns have a number of beneficial 
properties. First, the pattern is generated by replicating 
a relatively small kernel. This simplifies the mathemati- 
cal operations involved in demosaicing algorithms 
based on linear models that take into account various 
distortions introduced by lens, color responses, etc. 
[0027] Second, as any linear boundary in the image 
is traversed, the boundary repeatedly passes through 
color sensors of at least three linearly independent 
colors over a relatively short distance along the length 
of the boundary. This proyicles the data needed to dis- 
tinguish color variations from textural variations along 
the boundary. Since the human eye is particularly sen- 
sitive to linear edges, this feature provides substantial 
benefits. The distance that must be traversed along the 
linear boundary before passing over pixels of all of the 
different colors depends on the orientation of the linear 
boundary and the particular kernel. In the preferred em- 
bodiment, the kernel is chosen such that any line which 
passes through the center of a disk whose radius is five 
times the distance between the centers of two adjacent 
kernels will pass through at least three colors whose 
spectral responses are linearly independent within that 
disk, 

[0028] Third, within the body of the pattern, each pixel 
of any given color is adjacent to other pixels of that color. 
This provides the data needed to distinguish color vari- 
ations from textural variations within more general im- 
ages. 

[0029] Fourth, the kernel has equal numbers of color 
sensors of each color. This assures that all colors are 
sampled with the same density in the resulting CFA pat- 
tern. This, in turn, helps to ensure good image quality 
over a wide range of different scene illumination condi- 
tions, from incandescent lighting through to illuminants 
that are heavily biased toward the blue end of the spec- 
trum. 

[0030] The above embodiments of the present inven- 
tion have been described in terms of sensors for specific 
portions of the spectrum. However, it will be apparent to 
those skilled in the art that other choices of spectra! re- 
gions may be utilized without deviating from the teach- 
ings herein. 

(0031 ] Various modifications to the described embod- 
iments will be apparent to those skilled in the art from 
the foregoing description and accompanying drawings. 
[0032] The disclosures in United Stales patent appli- 
cation no. 09/009,426, from which this application 
claims priority, and in the abstract accompanying this 
application are incorporated herein by reference. 



Claims 

1 . Apparatus for recording an image, including a two- 
dimensional array of color image sensors, each 
color image sensor being operable to provide a 



measurement of a light intensity in a selected spec- 
tral region, said two-dimensional array comprising 
a plurality of identical blocks ( 1 2.22.32) of color sen- 
sors, said blocks (12,22.32) being juxtaposed to 

5 form said array each of said blocks (12.22.32) in- 
cluding equal numbers of sensors for each of said 
spectral regions, the number of different spectral re- 
gions being at least three, said sensors in said 
blocks (12.22.32) being arranged in a two-dimen- 

^0 slonal array having a plurality of rows and columns, 
wherein said sensors in said blocks (12,22,32) are 
arranged such that aoy straight line passing through 
a first sensor passes through sensors of at least 
three different colors whose spectral responses are 

?5 linearly independent. 

2. Apparatus as in Claim 1, wherein said straight line 
passes through said sensors of at least three differ- 
ent colors within a disk, centered at the first sensor 

20 and having a radius not larger than five limes the 
center to center spacing of said blocks (12,22.32) 
of sensors in said two-dimensional array of sensors. 

3. Apparatus as in Claim 1 or 2, wherein each of said 
2S sensors corresponding to one of said selected 

spectral regions in btocks (12.22.32) not adjacent 
to an edge of said two-dimensional array is adjacent 
a sensor corresponding to the same spectral re- 
gion. 

30 

4. Apparatus as in Claim 1 .2 or 3. wherein the number 
of different spectral regions is three, said spectral 
regions being denoted by R, G, and B; and said sen- 
sors in said blocks (12) are arranged in the following 

3S pattern or in an arrangement comprising cyclic per- 
mutations of the rows or columns of said pattern: 
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45 5. Apparatus as in Claim 1,2 or 3, wherein the number 
of different spectral regions is four, said spectral re- 
gions being denoted by C. M, Y, and G; and said 
sensors in said blocks (32) are arranged in the fol- 
lowing pattern, or in cyclic permutations of the rows 

5^ or columns of said pattern: 
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(57) Apparatus (10.20.30) for recording an image, 
includes a two-dimensional array of image sensors. 
Each image sensor provides a measurement of the light 
intensity in a selected spectral region. The two-dimen- 
sional array is generated from a plurality of identical 
blocks (12,22,32) of sensors, the blocks (1 2,22,32) be- 
ing juxtaposed to form the array. Each of the blocks 
( 1 2.22. 32) has equal n umbers of sensors for each of the 
spectral regions, the number of different spectral re- 
gions being at least three. The sensors in the blocks 
(1 2,22, 32) are arranged in a two-dimensional array hav- 
ing a plurality of rows and colunms. The sensors in the 
blocks (12,22.32) are arranged such that any straight 
line passing through a given sensor also passes through 
sensors of at least three different colors whose spectral 
responses are all linearly independent. All three such 
sensors may lie within a disk, centered at the first sensor 
and having a radius no larger than five times the center 
to center spacing of the blocks (12,22,32) in the sensor 
array Each of the sensors corresponding to one of the 
selected spectral regions in blocks (12,22.32) not adja- 
cent to an edge of the two-dimensional array may be 
adjacent a sensor corresponding to the same spectral 
region. 
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